Skip to content

feat: decompose patch 'other' error bucket into actionable sub-categories (Closes #1501) - #1502

Merged
Lexus2016 merged 1 commit into
mainfrom
evolution/issue-1501-patch-error-decomp
Jul 29, 2026
Merged

feat: decompose patch 'other' error bucket into actionable sub-categories (Closes #1501)#1502
Lexus2016 merged 1 commit into
mainfrom
evolution/issue-1501-patch-error-decomp

Conversation

@Lexus2016

Copy link
Copy Markdown
Owner

Automated evolution PR for issue #1501.

Problem

The patch tool's other error bucket accounts for 68 failures/week (87% of all patch errors). These failures are classified as unknown by classify_error() and other by _classify_failure_reason(), giving the model no actionable recovery path — leading to retry spirals on unknown causes.

Solution

Decompose the other/unknown bucket into actionable sub-categories:

  1. old_string_emptyclassify_error() now recognizes "old_string cannot be empty" (the error from fuzzy_find_and_replace line 73) and classifies it as old_string_empty instead of falling through to unknown.

  2. indentation_mismatch — a secondary classification pass in format_structured_error() that triggers when the primary classifier returns no_match. It finds the best-matching content line, then compares leading whitespace across ALL corresponding lines (the mismatch is often on a subsequent line, not the anchor). When the text matches but indentation differs (tabs vs spaces, different indent levels), it reclassifies as indentation_mismatch.

  3. Recovery hints — both new types get specific recovery suggestions: old_string_empty tells the model to provide a non-empty old_string; indentation_mismatch tells it to use read_file to see exact whitespace and update old_string to match.

  4. Introspection sub-categories_classify_failure_reason() in scripts/introspection_extract.py gets three new regex patterns (_PATCH_OLD_STRING_EMPTY_RE, _PATCH_INDENTATION_MISMATCH_RE, _PATCH_AMBIGUOUS_RE) checked BEFORE the generic _NO_MATCH_RE, so specific patch failures are tagged patch-old-string-empty, patch-indentation-mismatch, or patch-ambiguous instead of falling into other.

Files changed

  • tools/fuzzy_match.py — new ERROR_TYPES entries, classify_error sub-categories, format_structured_error secondary classification + recovery hints
  • scripts/introspection_extract.py — new regex patterns + classification chain entries
  • tests/tools/test_patch_self_correction.py — tests for new classify_error and format_structured_error paths

Validation

  • Lint: ✓ (ruff check)
  • Tests: ✓ (160 passed — test_patch_self_correction + test_introspection_extract + test_fuzzy_match + test_patch_failure_tracking)
  • Diff: 128 insertions, 0 deletions (within 200-line merge cap)

Closes #1501

…ries

Add old_string_empty and indentation_mismatch sub-categories to
classify_error() and ERROR_TYPES in tools/fuzzy_match.py, with
recovery hints in format_structured_error(). The indentation_mismatch
detection uses a secondary classification pass that compares leading
whitespace across corresponding lines when the primary classifier
returns no_match — catching the common case where the model's
old_string text matches but indentation (tabs vs spaces, indent level)
differs.

Update scripts/introspection_extract.py _classify_failure_reason()
with regex patterns for the new sub-categories (patch-old-string-empty,
patch-indentation-mismatch, patch-ambiguous), checked before the
generic _NO_MATCH_RE so specific patch failures are no longer swallowed
by the broad 'no match' or 'other' buckets.

This shrinks the 68 failures/week 'other' bucket (87% of patch errors)
by giving the model precise reasons and recovery paths.

Closes #1501

Co-Authored-By: Hermes Evolution <evolution@hermes.ai>
@github-actions github-actions Bot added bug Something isn't working introspection Found by session introspection labels Jul 29, 2026
@Lexus2016
Lexus2016 merged commit f6a99e9 into main Jul 29, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working introspection Found by session introspection

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FIX] patch tool 'other' error bucket undiagnosed — 68 failures/week, 87% of patch errors, retry spirals on unknown cause

1 participant